New tool to serialize mata test cases to file. #9441

jrothfeder · 2025-11-07T21:50:31Z

No description provided.

* Scaffolding for new appdistribution MCP tool. * Refactor business logic out of the appdistribution CLI so that it can be used by an MCP tool. * Wire new appdistribution tool up to the business logic. * Fix linting errors. * Update src/appdistribution/distribution.ts Co-authored-by: gemini-code-assist[bot] <176961590+gemini-code-assist[bot]@users.noreply.github.com> --------- Co-authored-by: Jamie Rothfeder <[email protected]> Co-authored-by: gemini-code-assist[bot] <176961590+gemini-code-assist[bot]@users.noreply.github.com>

* Rename appdistribution directory to apptesting * Make variables consistent with directory rename. --------- Co-authored-by: Jamie Rothfeder <[email protected]>

…9280) * Use a datastructure to represent test devices rather than a string. * Update src/mcp/tools/apptesting/tests.ts Co-authored-by: gemini-code-assist[bot] <176961590+gemini-code-assist[bot]@users.noreply.github.com> * Update src/mcp/tools/apptesting/tests.ts Co-authored-by: gemini-code-assist[bot] <176961590+gemini-code-assist[bot]@users.noreply.github.com> * Pretty --------- Co-authored-by: Jamie Rothfeder <[email protected]> Co-authored-by: gemini-code-assist[bot] <176961590+gemini-code-assist[bot]@users.noreply.github.com>

* Add initial MCP prompt for running automated tests * Fix typos

* Create a on-off test and execute. * Can now create a on-off test. * Update src/appdistribution/client.ts Co-authored-by: gemini-code-assist[bot] <176961590+gemini-code-assist[bot]@users.noreply.github.com> * Update src/mcp/tools/apptesting/tests.ts Co-authored-by: gemini-code-assist[bot] <176961590+gemini-code-assist[bot]@users.noreply.github.com> * PR feedback * Separate test check in to a different tool so that gemini can orchestrate running and checking for completion. * Set the devices field to optional --------- Co-authored-by: Jamie Rothfeder <[email protected]> Co-authored-by: gemini-code-assist[bot] <176961590+gemini-code-assist[bot]@users.noreply.github.com>

… description (#9322) * Use the same default device that's used in the Console * Update prompt to support generating a test case when there is no test description passed

* Add custom auto-enablement for app testing * Address gemini code assist comments * Fix intersection bug * Fix issues with test

Co-authored-by: Jamie Rothfeder <[email protected]>

gemini-code-assist · 2025-11-07T21:50:53Z

Summary of Changes

Hello @jrothfeder, I'm Gemini Code Assist¹! I'm currently reviewing this pull request and will post my feedback shortly. In the meantime, here's a summary to help you and other reviewers quickly get up to speed!

This pull request significantly enhances the Firebase App Distribution functionality by integrating support for AI-guided testing. It introduces new data structures and client methods to handle AI instructions for tests, refactors core distribution logic into more manageable units, and provides a suite of new prompts and tools for users to define, run, and monitor these advanced automated tests. The changes aim to streamline the process of creating reliable and repeatable test cases for mobile applications.

Highlights

AI-Guided Testing Integration: Introduced new interfaces (AIInstruction, AIStep) and updated AppDistributionClient.createReleaseTest to support AI-guided test instructions, enabling more sophisticated automated testing scenarios.
Refactored App Distribution Logic: Extracted release upload and automated test result polling into dedicated upload and awaitTestResults functions within src/appdistribution/distribution.ts, improving modularity and reusability.
New App Testing Prompts and Tools: Added a new apptesting feature with associated prompts (run_test) and tools (run_tests, testcase_export, check_status) to facilitate the creation, execution, and management of AI-guided tests.
Improved Utility Functions: Enhanced parseIntoStringArray and parseTestDevices to make the file parameter optional, and extracted toAppName for better code organization.
Conditional YAML Serialization: Modified toYamlTestCases to conditionally include the id field only when testCase.name is present, preventing potential issues with missing data.

Using Gemini Code Assist

The full guide for Gemini Code Assist can be found on our documentation page, here are some quick tips.

Invoking Gemini

You can request assistance from Gemini at any point by creating a comment using either /gemini <command> or @gemini-code-assist <command>. Below is a summary of the supported commands on the current page.

Feature	Command	Description
Code Review	`/gemini review`	Performs a code review for the current pull request in its current state.
Pull Request Summary	`/gemini summary`	Provides a summary of the current pull request in its current state.
Comment	@gemini-code-assist	Responds in comments when explicitly tagged, both in pull request comments and review comments.
Help	`/gemini help`	Displays a list of available commands.

Customization

To customize Gemini Code Assist for GitHub experience, repository maintainers can create a configuration file and/or provide a custom code review style guide (such as PEP-8 for Python) by creating and adding files to a .gemini/ folder in the base of the repository. Detailed instructions can be found here.

Limitations & Feedback

Gemini Code Assist may make mistakes. Please leave feedback on any instances where its feedback is incorrect or counter productive. You can react with 👍 and 👎 on @gemini-code-assist comments. If you're interested in giving your feedback about your experience with Gemini Code Assist for Github and other Google products, sign up here.

You can also get AI-powered code generation, chat, as well as code reviews directly in the IDE at no cost with the Gemini Code Assist IDE Extension.

Review the Privacy Notices, Generative AI Prohibited Use Policy, Terms of Service, and learn how to configure Gemini Code Assist in GitHub here. Gemini can make mistakes, so double check it and use code with caution. ↩

gemini-code-assist

Code Review

This pull request introduces a new tool for serializing MATA test cases and adds functionality for AI-guided testing. The changes include significant refactoring of the app distribution logic, extracting common functionalities like upload and awaitTestResults into a shared distribution.ts module. This improves code structure and reusability. New MCP prompts and tools for app testing are added, gated by an experiment flag. The overall changes are well-structured and the refactoring is a good improvement. I've left a couple of minor suggestions to improve code clarity and fix some typos in a prompt.

gemini-code-assist · 2025-11-07T21:52:38Z

src/appdistribution/distribution.ts

+        if (releaseTestNames.size === 0) {
+          utils.logSuccess("Automated test(s) passed!");
+          return;
+        } else {
+          continue;
+        }


The else { continue; } block is redundant. The for...of loop will naturally continue to the next iteration if the preceding if condition is not met. Removing this else block will make the code slightly cleaner.

if (releaseTestNames.size === 0) { utils.logSuccess("Automated test(s) passed!"); return; }

gemini-code-assist · 2025-11-07T21:52:38Z

src/mcp/prompts/apptesting/run_test.ts

+  You should already know the value of \`appId\' and \`projectId\` from earlier (if you only know \`projectNumber\',
+  use the \`firebase_get_project\` tool to get \`projectId\`). \`packageName\` is the package name of the app we tested. 
+  The \`apptesting_run_test\` tool returns a response with field \`name\` in the form 
+  projects/{projectNumber}/apps/{appId}/releases/{releaseId}/tests/{releaseTestId}. Extract the values for \'releaseId\' 
+  and \`releaseTestId\` and use provide a link to the results in the Firebase Console in the format:
+  \`https://console.firebase.google.com/u/0/project/{projectId}/apptesting/app/android:{packageName}/releases/{releaseId}/tests/{releaseTestId}\`.
+
+  You can check the status of the test using the \`apptesting_check_status\` tool with \`release_test_name\' set to
+  the name of the release test returned by the \`run_test\` tool.


There are a few typos in this part of the prompt text. There are stray single quotes or incorrect backticks around appId, releaseId, and release_test_name. These should be corrected to ensure the prompt is parsed correctly by the language model.

Suggested change

You should already know the value of \`appId\' and \`projectId\` from earlier (if you only know \`projectNumber\',

use the \`firebase_get_project\` tool to get \`projectId\`). \`packageName\` is the package name of the app we tested.

The \`apptesting_run_test\` tool returns a response with field \`name\` in the form

projects/{projectNumber}/apps/{appId}/releases/{releaseId}/tests/{releaseTestId}. Extract the values for \'releaseId\'

and \`releaseTestId\` and use provide a link to the results in the Firebase Console in the format:

\`https://console.firebase.google.com/u/0/project/{projectId}/apptesting/app/android:{packageName}/releases/{releaseId}/tests/{releaseTestId}\`.

You can check the status of the test using the \`apptesting_check_status\` tool with \`release_test_name\' set to

the name of the release test returned by the \`run_test\` tool.

You should already know the value of \`appId\` and \`projectId\` from earlier (if you only know \`projectNumber\`,

use the \`firebase_get_project\` tool to get \`projectId\`). \`packageName\` is the package name of the app we tested.

The \`apptesting_run_test\` tool returns a response with field \`name\` in the form

projects/{projectNumber}/apps/{appId}/releases/{releaseId}/tests/{releaseTestId}. Extract the values for \`releaseId\`

and \`releaseTestId\` and use provide a link to the results in the Firebase Console in the format:

\`https://console.firebase.google.com/u/0/project/{projectId}/apptesting/app/android:{packageName}/releases/{releaseId}/tests/{releaseTestId}\`.

You can check the status of the test using the \`apptesting_check_status\` tool with \`release_test_name\` set to

the name of the release test returned by the \`run_test\` tool.

src/appdistribution/yaml_helper.ts

src/mcp/tools/apptesting/tests.ts

tagboola · 2025-11-10T14:37:48Z

src/mcp/tools/apptesting/tests.ts

 );

+<<<<<<< HEAD
+export const testcase_export = tool(


I don't know if we want to create a new tool for this. In general, the CLI team is trying to reduce the number of tools in the Firebase MCP server, which is why I ended up combining the check_tests and get_devices tools in the previous PR.

Jamie Rothfeder and others added 27 commits October 6, 2025 12:10

creation of feature branch

11b53b8

Merge branch 'master' into feature-branch/mcp/mobile-testing

7d8e857

Rename appdistribution directory to apptesting (#9268)

455d6d3

* Rename appdistribution directory to apptesting * Make variables consistent with directory rename. --------- Co-authored-by: Jamie Rothfeder <[email protected]>

Merge branch 'master' into feature-branch/mcp/mobile-testing

b26c98b

Merge branch 'master' into feature-branch/mcp/mobile-testing

62e3171

Merge branch 'master' into feature-branch/mcp/mobile-testing

26dd4f3

Merge branch 'master' into feature-branch/mcp/mobile-testing

c318b06

Add run_test prompt (#9292)

81c1a61

* Add initial MCP prompt for running automated tests * Fix typos

Merge branch 'master' into feature-branch/mcp/mobile-testing

75d279f

Merge branch 'master' into feature-branch/mcp/mobile-testing

9bbf17a

Use the same default device that's used in the Console (#9320)

973e8d7

Update prompt to support generating a test case when there is no test…

82e1724

… description (#9322) * Use the same default device that's used in the Console * Update prompt to support generating a test case when there is no test description passed

Merge branch 'master' into feature-branch/mcp/mobile-testing

c081d6d

Add custom auto-enablement for app testing (#9373)

380b001

* Add custom auto-enablement for app testing * Address gemini code assist comments * Fix intersection bug * Fix issues with test

Add get devices tool (#9387)

8a95353

Display link to results in the Firebase Console (#9406)

7356c41

Merge branch 'master' into feature-branch/mcp/mobile-testing

32c5a93

Place app testing tools behind an experiment

7f94828

Address GCA comments

cb9c42b

Explicitly set default devices

81a2734

Address PR comments

0bc8208

Merge branch 'master' into feature-branch/mcp/mobile-testing

d642542

Fix the status URL. (#9438)

9877ee5

Co-authored-by: Jamie Rothfeder <[email protected]>

Add a tool to export tests to a file.

1c4631d

github-project-automation bot added this to [Cloud] Extensions + Functions Nov 7, 2025

gemini-code-assist bot reviewed Nov 7, 2025

View reviewed changes

Merge branch 'master' into serizlize

f34e992

jrothfeder requested a review from tagboola November 7, 2025 21:53

kaibolay reviewed Nov 7, 2025

View reviewed changes

src/appdistribution/yaml_helper.ts Outdated Show resolved Hide resolved

src/mcp/tools/apptesting/tests.ts Outdated Show resolved Hide resolved

src/mcp/tools/apptesting/tests.ts Outdated Show resolved Hide resolved

src/mcp/tools/apptesting/tests.ts Outdated Show resolved Hide resolved

tagboola reviewed Nov 10, 2025

View reviewed changes

Jamie Rothfeder added 2 commits November 10, 2025 13:21

Remove merge conflict markers.

a1de3cd

Simplify conversion of testcases to yaml.

6d6b1a9

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Uh oh!

New tool to serialize mata test cases to file. #9441

New tool to serialize mata test cases to file. #9441

Uh oh!

jrothfeder commented Nov 7, 2025

Uh oh!

gemini-code-assist bot commented Nov 7, 2025

Uh oh!

gemini-code-assist bot left a comment

Uh oh!

gemini-code-assist bot Nov 7, 2025

Uh oh!

gemini-code-assist bot Nov 7, 2025

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

tagboola Nov 10, 2025

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

3 participants

New tool to serialize mata test cases to file. #9441

Are you sure you want to change the base?

New tool to serialize mata test cases to file. #9441

Uh oh!

Conversation

jrothfeder commented Nov 7, 2025

Uh oh!

gemini-code-assist bot commented Nov 7, 2025

Summary of Changes

Highlights

Footnotes

Uh oh!

gemini-code-assist bot left a comment

Choose a reason for hiding this comment

Code Review

Uh oh!

gemini-code-assist bot Nov 7, 2025

Choose a reason for hiding this comment

Uh oh!

gemini-code-assist bot Nov 7, 2025

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

tagboola Nov 10, 2025

Choose a reason for hiding this comment

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

3 participants